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A DATA-EFFICIENT AND SELF ADAPTING IMAGING 
SPECTROMETRY METHOD AND AN APPARATUS 

THEREOF 

5 FIELD OF THE INVENTION 

[0001] The present invention relates to a method and system for capturing 

an image of a scene and, more particularly to a data-efficient and self adapting 
method and system for analyzing spectral data for use in a data efficient and self 
adapting spectral imaging method and system. 

10 

BACKGROUND OF THE INVENTION 

[0002] Spectral imaging is based on capturing multiple channels of color 

data for each pixel in an image. Each channel is associated with a different 
spectral sensitivity signature. Spectral imaging with its multiple channels is 

15 distinguished from traditional imaging because the latter typically collects only 
three channels, usually red, green and blue. A properly calibrated spectral 
approach can maintain the reflectance or transmittance properties of scene objects 
or can detect the energy level of quanta which reach the detector for each pixel. 
Traditional imaging systems can not do this and are limited to describing color 

20 appearance of a scene under highly constrained environmental conditions. Thus, 
spectral imaging far exceeds traditional imaging in terms of flexibility, power and 
the ability to analyze scene contents. 

[0003] Since spectral imaging systems are based on capturing multiple 

25 channels of color data, they capture and require the processing and storage of far 
more data than traditional imaging systems. As a result, spectral imaging systems 
are faced with data overload problems. 
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[0004] A method for spectral imaging in accordance with one embodiment 

of the present invention includes capturing high spectral resolution data from at 
least a portion of a scene using a plurality of channels, determining a first set of 
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channels from the plurality of channels and an associated first set of parameters to 
a transform which can reconstruct spectra of the portion of the scene to within a 
first error tolerance from the captured high spectral resolution data, and capturing 
an image of the scene using the first set of channels from which original scene 
5 reflectance, transmittance or radiance may be estimated at high spectral resolution 
and high spatial resolution when the transform using the first set of parameters is 
applied to the captured image. 



[0005] A system for spectral imaging in accordance with one embodiment 

^ 10 of the present invention includes an imaging system, a spectral processing system 

jfi and a data storage medium. The imaging system captures high spectral resolution 

data from at least a portion of a scene using a plurality of channels. The spectral 
PJ processing system determines a first set of channels from the plurality of channels 

m 

fy and an associated first set of parameters to a transform which can reconstruct 

^ 15 spectra of the portion of the scene to within a first error tolerance from the 

M 1 captured high spectral resolution data, wherein the imaging system captures high 

spatial resolution image data of the scene using the first set of channels, and stores 
in the data storage medium the image data and the first set of transform 
y, parameters. Subsequently, the spectral processing system may retrieve the first set 

20 of parameters and image data from the digital storage medium and process the 
image data in accordance with the first set of transform parameters, resulting in 
estimates of the original scene reflectance, transmittance or radiance at high 
spectral resolution for each pixel of the image. 

25 [0006] The present invention improves data overload problems previously 

associated with general spectral imaging as well as alleviating the tradeoffs 
between accuracy and generality associated with previous specialized spectral 
imaging. The present invention has recognized that for any particular object in a 
scene or for any scene in total there is one or more sets of channels which contain 

30 an optimally minimum number of channels and an associated transform which can 
be used for accurate spectral reconstruction of that object or scene. As a result, 
the present invention strives to approximate an optimal set of channels and to 
derive an optimal transform for every scene or portion of scene encountered. The 
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extent to which optimums are realized is limited by the specifics of any particular 
system implementation. The number of channels in an optimal set could be as few 
as two or three or many more, depending upon the specific characteristics of an 
object or scene and the nature of the mathematical constructs of the transforms 
5 chosen for an implementation. 
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[0007] By addressing the data overload problem associated with the large 

number of channels needed with prior general spectral imaging systems, the 
present invention is able to increase the speed of data capture. Since the number 
10 of channels to be captured is limited, the amount of data to be moved through the 
system is likewise reduced allowing for faster imaging and faster subsequent 
image processing. 



fH [0008] Additionally, by reducing the spectral data being captured in each 

til 

I 15 image, the present invention has a higher storage capacity for images and lower 
^j' bandwidth demands. Image compression requirements are reduced accordingly. 



[0009] Further, since fewer channels are needed with the present 

invention, system complexity is reduced, system specifications are relaxed, 
20 manufacturing yield goes up all resulting in lower system cost. 



[0010] Another advantage of the present invention is that it is both data 

efficient and general since it automatically chooses an appropriate low bandwidth 
configuration for every scene it encounters. Prior specialized spectral imaging 
25 systems with low bandwidth were fixed with a single configuration and thus for 
high accuracy they were restricted to imaging one particular class of objects for 
which they were customized. 
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[0011] FIG. 1 is a block diagram of a spectral imaging system in 

accordance with the present invention; 



[0012] FIG. 2 is a flow chart of a method for spectral imaging in 

accordance with a first embodiment of the present invention; 
FIG. 3 is a flow chart of a method for spectral imaging in accordance with a 
second embodiment of the present invention; 

[0013] FIG. 4 is a flow chart of a method for spectral imaging in 

accordance with a third embodiment of the present invention; and 

[0014] FIG. 5 is a flow chart of a method for spectral imaging in 

accordance with a fourth embodiment of the present invention. 

DETAILED DESCRIPTION 

[0015] A system 10 and method for the capture of still spectral images in 

accordance with one embodiment of the present invention is illustrated in FIGS. 1 
and 2. The system 10 includes an imaging system 12 and a spectral processing 
system 14. The method includes capturing high spectral resolution data from at 
least a portion of a scene using a plurality of channels, determining a first set of 
channels from the plurality of channels and associated set of parameters to a first 
transform which can reconstruct spectra of the portion of the scene to within a first 
error tolerance from the captured high spectral resolution data, and capturing an 
image of the scene using the set of channels from which original scene 
reflectance, transmittance or radiance may be estimated at high spectral resolution 
and high spatial resolution when the transform using the set of parameters is 
applied to the captured image. The present invention improves data overload 
problems previously associated with spectral imaging by only taking the optimally 
minimum number of channels for every scene and adjusting the optimal set of 
channels being taken intermittently. 

[0016] Referring more specifically to FIG. 1, in this particular 

embodiment the imaging system 12 is a spectral imaging system that captures 
multiple channels of color data for each pixel in an image, although other types of 
imaging systems can be used. The imaging system is coupled to and receives 



information and instructions, such as the particular channels of color data to 
capture, from the spectral processing imaging system 14. The imaging system 12 
also transmits information and instructions, such as captured spectral data of an 
object or portion of a scene, to the spectral processing system 14. In this 
particular embodiment, the imaging system 12 comprises interference filters 
positioned in front of a photodetector or other imager to capture the channel data 
for the set of channels, set a, although imaging system 12 can comprise other 
components, such as absorption filters positioned in front of a photodetector or 
other imager, a tunable filter positioned in front of a photodetector or other 
imager, or a rewritable filter positioned in front of a photodetector or other imager. 
The particular channel being captured by the tunable filter or the rewritable filter 
can be synthesized by time and wavelength modulation of the tunable filter or the 
rewritable filter. 

[0017] The spectral processing system 14 includes a central processing 

unit (CPU) 16, a memory 18, an input/output interface 20, and a user input device 
22 which are coupled together by a bus system 19 or other link, although the 
spectral processing system may comprise other components, other numbers of the 
components, and other combinations of the components. The processor executes a 
program of stored instructions for the method for data efficient and self adapting 
spectral imaging in accordance with the present invention as described herein and 
illustrated in FIGS. 2 and 3. In this particular embodiment, those programmed 
instructions are stored in memory 18, although some or all could be stored and 
retrieved from other locations. A variety of different types of memory storage 
devices, such as a random access memory (RAM) or a read only memory (ROM) 
in the system or a floppy disk, hard disk, CD ROM, or other computer readable 
medium which is read from and/or written to by a magnetic, optical, or other 
reading and/or writing system that is coupled to the processor 16, can be used for 
memory 1 8. The input/output interface 20 is used to operatively couple and 
communicate between the spectral processing system 14 and the imaging system 
12, the reproduction system 24, and the storage system 26. A variety of different 
types of connections and communication techniques can be used to transmit 
signals between the spectral processing system 14 and the imaging system 12, the 
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reproduction system 24, and the storage system 26. The user input device 22 
enables an operator to generate and transmit signals or commands to the processor 
16, such as error tolerances A variety of different types of user input devices 22 
can be used, such as a keyboard or computer mouse. 

[0018] In this particular embodiment, reproduction system 24 is coupled to 

the spectral processing system 14 and is another imaging system, such as a printer 
or a display device, although other types of systems could be used. The 
reproduction system 24 recreates the captured image based on the estimated 
spectra reconstructed by applying a parameterized transform to the channels of 
image data for every portion of a scene received from the spectral processing unit. 

[0019] In this particular embodiment, the digital data storage medium 26 is 

coupled to the spectral processing system 14 and is a memory storage device, such 
as a random access memory (RAM) or a read only memory (ROM) in the system 
or a floppy disk, hard disk, CD ROM, or other computer readable medium which 
is read from and/or written to by a magnetic, optical, or other reading and/or 
writing system, in a remote computer processing system. The storage system 26 
can store spectral data from the spectral processing system 14 for retrieval and use 
at a later time. 

[0020] System 10 may also include a secondary imaging system 27 that is 

a spectral imaging system with very little spatial extent, but very high spectral 
resolution, although other types of imaging systems configured in other manners 
25 can be used. The secondary imaging system 27 is coupled to and receives 

information and/or instructions from and transmits information and/or instructions 
to the spectral processing imaging system 14. Although an optional secondary 
imaging system 27 is shown, the primary imaging system 12 could also perform 
the functions of the secondary imaging system 27, so that only one imaging 
30 system is needed. 

[0021] Referring to FIG. 2, a method for spectral imaging that is data 

efficient and self adapting in accordance with a first embodiment of the present 
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invention will be described. In step 30, a portion or part of a scene using all 
available channels is captured by the imaging system 12 and transmitted to the 
spectral processing system 14. Typically, the imaging system 12 will have a 
plethora of channels available to it. The size of the portion of the scene captured 
can vary as needed or desired for the particular application. 

[0022] In step 31, the spectral processing system 14 transforms the 

channel data captured by the primary imaging system 12 to spectra using a 
transform to. In this example, transform to was previously derived and stored in 
memory 1 8, although in other implementations it can be made known to the 
system 10 through other means, such as in a user initiated calibration step. By 
applying transform to to the channel data captured in step 30, highly accurate 
spectra for each pixel in the captured portion of the scene is derived. 

[0023] In step 32, the spectral processing system 14 determines an 

optimally minimum set of channels which are referred to as, "set a" in this 
particular example. The spectral processing system 14 has a stored error 
tolerance, e, and also determines a second transform t which is used to transform 
the data to spectra for imaging. The spectral processing system 14 performs an 
iterative optimization determining a matrix which when applied to the captured 
pixel data from a set of channels, the closest reconstruction of the highly accurate 
spectra may be realized. The set with the smallest number of channels which can 
approximate the highly accurate spectra across all captured pixels within an 
average spectral RMS difference that is less them the error tolerance e is chosen as 
the optimal set. Transform t includes the derived matrix associated with this 
optimally minimum set of channels, set a. The spectral processing system 14 
transmits instructions to imaging system 12 to capture the image using the 
optimally minimum set of channels, set a. 

[0024] In step 34, the primary imaging system 14 captures the entire scene 

using the optimally minimum channel set, set a. The imaging system 12 transmits 
the multiple channel data captured using the optimally minimum set of channels, 
set a, to the spectral processing system 14. In step 35, the spectral processing 
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system 14 stores the channel data and the transform t in storage 26, although the 
data and transform can be stored elsewhere, such as in memory 1 8. In step 37, the 
process ends. 

[0025] A system 10 and method for spectral imaging in accordance with a 

second embodiment of the present invention is identical to the one described 
above and illustrated in FIGS. 1 and 2, except as described below and illustrated 
in FIG. 3. This second embodiment is more robust than the first embodiment of 
the present invention described above because it has the ability to update its filter 
and transform choices as it images a scene. 

[0026] Steps 30-32 are the same as described above with reference to FIG. 

2. In step 34' in FIG. 3 in this particular embodiment, the imaging system 14 
captures a portion of the scene using the optimally minimum channel set, set a. 
The size of the image taken for this portion of the scene can vary as needed or 
desired by the particular application. The imaging system 12 transmits the 
multiple channel data captured using the optimally minimum set of channels, set 
a, to the spectral processing system 14. 

[0027] In step 36, the spectral processing system 14 decides whether to 

check the results of the captured set of spectral data and the associated transform. 
In this particular embodiment, the check is carried out periodically at intervals 
entered in by the operator using user input device 20 or from preset intervals 
stored in instructions in memory 1 8, although other systems and methods for 
checking can be used, such as checking the captured spectral data randomly or at 
the request of an operator using the user input device 22. 

[0028] If a check is not going to be performed by the spectral processing 

system 14 in step 36, then the No branch is taken to step 38. In step 38, the 
30 spectral processing system 14 stores the spectral data for the portion of the image 
or scene captured and the associated transform, t, in memory 18, although this 
information can be stored elsewhere, such as in storage system 26 or in 
reproduction system 24. 
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[0029] In step 40, the spectral processing system 14 determines whether 

the spectral imaging of the particular scene or scenes to be captured has ended. 
By way of example only, the operator using the user input device 22 may signal 
the spectral processing system 14 that spectral image capturing is completed. If 
the spectral imaging has ended, then the Yes branch is taken to step 37 where the 
method ends. 

[0030] If the spectral imaging has not ended, then the No branch is taken 

from step 40 to step 42 where the imaging system 12 captures spectral data for the 
next portion of the scene using the current optimally minimum set of channels, set 
a. Once data for the next portion of the scene has been captured and transmitted to 
the spectral processing system 14, then the method goes back to step 36 to decide 
whether to check the results of the captured set of spectral data and the associated 
transform. Again, if a check is not going to be performed in step 36, then the No 
branch is taken to step 38 as described above. 

[0031] If a check is going to be performed by the spectral processing 

system 14 in step 36, then the Yes branch is taken to step 44. In step 44, the 
spectral processing system 44 converts the captured data f rom the current 
optimally minimum set of channels, set a, for a current portion of the scene to 
estimate spectral data using the associated transform, t. 

[0032] In step 46, the spectral processing system 14 instructs the imaging 

system 12 to capture the current portion of the scene using the full set of available 
channels. The imaging system 12 captures and transmits the data for the current 
portion using the full set of channels to the spectral processing system 14. 

[0033] In step 47, the spectral processing system 14 uses the first 

transform, transform to, to create a highly accurate estimate of spectral data of this 
portion of the scene from the channel data captured using the full set of channels. 
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[0034] In step 48, the spectral processing system 14 compares these two 

estimates of spectral data through calculating an error which is the average RMS 
difference across all pixels captured in step 46. 

[0035] In step 50, the operator can enter in a tolerance, e, for the error 

using the user input device 22 or the tolerance can be retrieved from a memory, 
such as memory 18, in spectral processing system 14. In step 52, the spectral 
processing system 14 determines whether the error is less than the tolerance e. If 
the error is less than the tolerance e, than the Yes branch is taken back to step 38 
as described earlier and the spectral processing system 14 continues to use the 
current optimally minimum set of channels, set a, and the associated current 
transform, t. 

[0036] If the error is not less than the tolerance e, then the No branch is 

taken back to step 54. In step 54, the spectral processing system 14 determines an 
updated optimal minimum set of channels which are assigned to set a and 
determines an updated transform, t n . If an updated transform, t n , is derived then 
the previous transform t is stored in such a way that the spatial relationship 
between it and prior captured image pixels is preserved. 

[0037] Once step 54 is completed, then next the method and system go 

back to step 38 which was described earlier, except that the spectral processing 
system 14 now uses the updated optimally minimum set of channels, set a, and the 
associated updated transform, t n . The new parameters will continue to be used in 
subsequent imaging until error conditions are found which call for yet another 
channel/transformation setup as described herein. 

[0038] A system 10 and method for spectral imaging in accordance with a 

third embodiment of the present invention is identical to the one described above 
and illustrated in FIGS. 1 and 3, except as described below and illustrated in FIG. 
4. This particular embodiment describes the use of the present invention in a 
method for spectral video or spectral cinema configuration. This particular 
embodiment also intermittently checks to see if a high level of spectral accuracy is 



being realized. If insufficient accuracy is being maintained a new set of channels 
and a new transform is used in the primary system. 

[0039] Referring to FIG. 4, in step 30' a portion or part of a scene using all 

available channels is captured by the secondary imaging system 27 and 
transmitted to the spectral processing system 14. Typically, the imaging system 
27 will have a plethora of channels available to it. The size of the portion of the 
scene captured can vary as needed or desired for the particular application. 

[0040] In step 31', the spectral processing system 14 transforms the 

channel data captured by the secondary imaging system 27 to spectra using a 
transform to. In this example, transform to was previously derived and stored in 
memory 1 8, although in other implementations it can be made known to the 
system 10 through other means, such as in a user initiated calibration step. By 
applying transform to to the channel data captured in step 30% highly accurate 
spectra for each pixel in the captured portion of the scene is derived. 

[0041] In step 32', the spectral processing system 14 determines an 

optimally minimum set of channels from the secondary imaging system 27 which 
are referred to as, "set a" in this particular example. The spectral processing 
system 14 has a stored error tolerance, e, and also determines a second transform t 
which is used to transform the data to spectra for imaging. The spectral 
processing system 14 performs an iterative optimization determining a matrix 
which when applied to the captured pixel data from a set of channels, the closest 
reconstruction of the highly accurate spectra may be realized. The set with the 
smallest number of channels which can approximate the highly accurate spectra 
across all captured pixels within an average spectral RMS difference that is less 
than the error tolerance e is chosen as the optimal set. Transform t includes the 
derived matrix associated with this optimally minimum set of channels, set a. The 
spectral processing system 14 transmits instructions to imaging system 12 to 
capture the image using the optimally minimum set of channels, set a. 
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[0042] In step 42', the imaging system 12 continues to captures spectral 

data for scene using the current optimally minimum set of channels, set a. Since 
the other steps in FIG. 4 with like numbers to those in FIG. 3 are identical to those 
steps as described earlier, they will not be discussed again here. 

[0043] A system 10 and method for spectral imaging in accordance with a 

fourth embodiment of the present invention is also identical to the one described 
above and illustrated in FIGS. 1 and 2, except as described below and illustrated 
in FIG. 5. This particular embodiment describes a method for capture of still 
spectral images where there is a primary imaging system 12 with a fixed set of 
channels available for capture at all times and a secondary imaging system 27 of 
little spatial extent, but with high spectral resolution. Unlike the embodiment 
described in FIG. 3, the primary imaging system 12 is not configurable in this 
particular embodiment and it is only the transform to spectra which can be 
updated periodically 

[0044] Referring to FIG. 5, in step 32" the spectral processing system 14 

determines a second transform t which is used to transform the channel data to 
spectra for imaging. In step 35', the spectral processing system 14 stores the 
transform t in storage 26, although the transform can be stored elsewhere, such as 
in memory 18. Since the other steps in FIG. 5 with like numbers to those in FIG. 
2 are identical to those steps as described earlier, they will not be discussed again 
here. 

[0045] As these particular embodiments illustrate, the present invention 

improves data overload problems previously associated with spectral imaging by 
only taking the optimal minimal set of data for every object in or portion of a 
scene. By reducing the data, the system 10 and method are faster and lower cost 
than prior spectral imaging systems and have lower bandwidth demands. Further, 
the present invention has the self-adapting capability to react to the scene it is 
imaging or to adapt to trends detected over time from scenes it has been imaging, 
in order to select optimal minimum channel sets or to make optimal use of its 
given channels for data-efficient scene spectral reconstruction. 



-13- 



[0046] The present invention can be used in a variety of different types of 

applications, such as scanning images of fine-arts paintings or for capturing 
spectral moving images. Color reproduction systems that incorporate the present 
invention would benefit from the capture of spectral information because they can 
accurately simulate the effect of changing lighting in the taking environment or 
can overcome problems in changing lighting in the viewing environment. In 
many potential applications of the present invention, through spectral analysis, the 
chemical composition of points in a scene can be assessed creating opportunities 
for the exploitation of this information. For example the present invention could 
be used in a consumer camera system. Here, spectral analysis of the scene would 
enable the location of important object classes such as faces, synthetic clothing, 
etc. all of which could be of use in a secondary analysis. Among uses for such 
secondary analysis could be the improvement of final image quality or could be 
the cataloging of images according to scene content. Other examples include 
commercial ID systems and surveillance systems that would likewise have new 
capabilities because image objects could be segmented according to spectral 
signatures. In another example, the present invention could be used in a motion 
picture film restoration system. Taking advantage of prior knowledge of expected 
material composition and deterioration characteristics for motion picture film, the 
spectral information captured by the system 10 as described above could be 
subsequently analyzed to determine which spectral aspects were signatures of 
original colorant levels and which spectral aspects were signatures of deterioration 
artifacts to be removed during the restoration process. 

[0047] Having thus described the basic concept of the invention, it will be 

rather apparent to those skilled in the art that the foregoing detailed disclosure is 
intended to be presented by way of example only, and is not limiting. Various 
alterations, improvements, and modifications will occur and are intended to those 
skilled in the art, though not expressly stated herein. These alterations, 
improvements, and modifications are intended to be suggested hereby, and are 
within the spirit and scope of the invention. 



